Evaluation of bias and gender/racial concordance based on sentiment analysis of narrative evaluations of clinical clerkships using natural language processing.

There is increasing interest in understanding potential bias in medical education. We used natural language processing (NLP) to evaluate potential bias in clinical clerkship evaluations. Data from medical evaluations and administrative databases for medical students enrolled in third-year clinical clerkship rotations across two academic years. We collected demographic information of students and faculty evaluators to determine gender/racial concordance (i.e., whether the student and faculty identified with the same demographic). We used a multinomial log-linear model for final clerkship grades, using predictors such as numerical evaluation scores, gender/racial concordance, and sentiment scores of narrative evaluations using the SentimentIntensityAnalyzer tool in Python. 2037 evaluations from 198 students were analyzed. Statistical significance was defined as P < 0.05. Sentiment scores for evaluations did not vary significantly by student gender, race, or ethnicity (P = 0.88, 0.64, and 0.06, respectively). Word choices were similar across faculty and student demographic groups. Modeling showed narrative evaluation sentiment scores were not predictive of an honors grade (odds ratio [OR] 1.23, P = 0.58). Numerical evaluation average (OR 1.45, P < 0.001) and gender concordance between faculty and student (OR 1.32, P = 0.049) were significant predictors of receiving honors. The lack of disparities in narrative text in our study contrasts with prior findings from other institutions. Ongoing efforts include comparative analyses with other institutions to understand what institutional factors may contribute to bias. NLP enables a systematic approach for investigating bias. The insights gained from the lack of association between word choices, sentiment scores, and final grades show potential opportunities to improve feedback processes for students.

A scoping review of artificial intelligence in medical education: BEME Guide No. 84.

BACKGROUND: Artificial Intelligence (AI) is rapidly transforming healthcare, and there is a critical need for a nuanced understanding of how AI is reshaping teaching, learning, and educational practice in medical education. This review aimed to map the literature regarding AI applications in medical education, core areas of findings, potential candidates for formal systematic review and gaps for future research. METHODS: This rapid scoping review, conducted over 16 weeks, employed Arksey and O'Malley's framework and adhered to STORIES and BEME guidelines. A systematic and comprehensive search across PubMed/MEDLINE, EMBASE, and MedEdPublish was conducted without date or language restrictions. Publications included in the review spanned undergraduate, graduate, and continuing medical education, encompassing both original studies and perspective pieces. Data were charted by multiple author pairs and synthesized into various thematic maps and charts, ensuring a broad and detailed representation of the current landscape. RESULTS: The review synthesized 278 publications, with a majority (68%) from North American and European regions. The studies covered diverse AI applications in medical education, such as AI for admissions, teaching, assessment, and clinical reasoning. The review highlighted AI's varied roles, from augmenting traditional educational methods to introducing innovative practices, and underscores the urgent need for ethical guidelines in AI's application in medical education. CONCLUSION: The current literature has been charted. The findings underscore the need for ongoing research to explore uncharted areas and address potential risks associated with AI use in medical education. This work serves as a foundational resource for educators, policymakers, and researchers in navigating AI's evolving role in medical education. A framework to support future high utility reporting is proposed, the FACETS framework.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

PURPOSE: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. METHOD: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. RESULTS: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. CONCLUSIONS: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Considering the Secondary Use of Clinical and Educational Data to Facilitate the Development of Artificial Intelligence Models.

ABSTRACT: Medical training programs and health care systems collect ever-increasing amounts of educational and clinical data. These data are collected with the primary purpose of supporting either trainee learning or patient care. Well-established principles guide the secondary use of these data for program evaluation and quality improvement initiatives. More recently, however, these clinical and educational data are also increasingly being used to train artificial intelligence (AI) models. The implications of this relatively unique secondary use of data have not been well explored. These models can support the development of sophisticated AI products that can be commercialized. While these products have the potential to support and improve the educational system, there are challenges related to validity, patient and learner consent, and biased or discriminatory outputs. The authors consider the implications of developing AI models and products using educational and clinical data from learners, discuss the uses of these products within medical education, and outline considerations that should guide the appropriate use of data for this purpose. These issues are further explored by examining how they have been navigated in an educational collaborative.

Trainees' Perspectives on the Next Era of Assessment and Precision Education.

ABSTRACT: The next era of assessment in medical education promises new assessment systems, increased focus on ensuring high-quality equitable patient care, and precision education to drive learning and improvement. The potential benefits of using learning analytics and technology to augment medical training abound. To ensure that the ideals of this future for medical education are realized, educators should partner with trainees to build and implement new assessment systems. Coproduction of assessment systems by educators and trainees will help to ensure that new educational interventions are feasible and sustainable. In this paper, the authors provide a trainee perspective on 5 key areas that affect trainees in the next era of assessment: (1) precision education, (2) assessor education, (3) transparency in assessment development and implementation, (4) ongoing evaluation of the consequences of assessment, and (5) patient care data as sources of education outcomes.As precision education is developed, it is critical that trainees understand how their educational data are collected, stored, and ultimately utilized for educational outcomes. Since assessors play a key role in generating assessment data, it is important that they are prepared to give high-quality assessments and are continuously evaluated on their abilities. Transparency in the development and implementation of assessments requires communicating how assessments are created, the evidence behind them, and their intended uses. Furthermore, ongoing evaluation of the intended and unintended consequences that new assessments have on trainees should be conducted and communicated to trainees. Finally, trainees should participate in determining what patient care data are used to inform educational outcomes. The authors believe that trainee coproduction is critical to building stronger assessment systems that utilize evidence-based educational theories for improved learning and ultimately better patient care.

Remote learning developments in postgraduate medical education in response to the COVID-19 pandemic - A BEME systematic review: BEME Guide No. 71.

BACKGROUND: Prior reviews investigated medical education developments in response to COVID-19, identifying the pivot to remote learning as a key area for future investigation. This review synthesized online learning developments aimed at replacing previously face-to-face 'classroom' activities for postgraduate learners. METHODS: Four online databases (CINAHL, Embase, PsychINFO, and PubMed) and MedEdPublish were searched through 21 December 2020. Two authors independently screened titles, abstracts and full texts, performed data extraction, and assessed risk of bias. The PICRAT technology integration framework was applied to examine how teachers integrated and learners engaged with technology. A descriptive synthesis and outcomes were reported. A thematic analysis explored limitations and lessons learned. RESULTS: Fifty-one publications were included. Fifteen collaborations were featured, including international partnerships and national networks of program directors. Thirty-nine developments described pivots of existing educational offerings online and twelve described new developments. Most interventions included synchronous activities (n Fif5). Virtual engagement was promoted through chat, virtual whiteboards, polling, and breakouts. Teacher's use of technology largely replaced traditional practice. Student engagement was largely interactive. Underpinning theories were uncommon. Quality assessments revealed moderate to high risk of bias in study reporting and methodology. Forty-five developments assessed reaction; twenty-five attitudes, knowledge or skills; and two behavior. Outcomes were markedly positive. Eighteen publications reported social media or other outcomes, including reach, engagement, and participation. Limitations included loss of social interactions, lack of hands-on experiences, challenges with technology and issues with study design. Lessons learned highlighted the flexibility of online learning, as well as practical advice to optimize the online environment. CONCLUSIONS: This review offers guidance to educators attempting to optimize learning in a post-pandemic world. Future developments would benefit from leveraging collaborations, considering technology integration frameworks, underpinning developments with theory, exploring additional outcomes, and designing and reporting developments in a manner that supports replication.

An update on developments in medical education in response to the COVID-19 pandemic: A BEME scoping review: BEME Guide No. 64.

BACKGROUND: COVID-19 has fundamentally altered how education is delivered. Gordon et al. previously conducted a review of medical education developments in response to COVID-19; however, the field has rapidly evolved in the ensuing months. This scoping review aims to map the extent, range and nature of subsequent developments, summarizing the expanding evidence base and identifying areas for future research. METHODS: The authors followed the five stages of a scoping review outlined by Arskey and O'Malley. Four online databases and MedEdPublish were searched. Two authors independently screened titles, abstracts and full texts. Included articles described developments in medical education deployed in response to COVID-19 and reported outcomes. Data extraction was completed by two authors and synthesized into a variety of maps and charts. RESULTS: One hundred twenty-seven articles were included: 104 were from North America, Asia and Europe; 51 were undergraduate, 41 graduate, 22 continuing medical education, and 13 mixed; 35 were implemented by universities, 75 by academic hospitals, and 17 by organizations or collaborations. The focus of developments included pivoting to online learning (n = 58), simulation (n = 24), assessment (n = 11), well-being (n = 8), telehealth (n = 5), clinical service reconfigurations (n = 4), interviews (n = 4), service provision (n = 2), faculty development (n = 2) and other (n = 9). The most common Kirkpatrick outcome reported was Level 1, however, a number of studies reported 2a or 2b. A few described Levels 3, 4a, 4b or other outcomes (e.g. quality improvement). CONCLUSIONS: This scoping review mapped the available literature on developments in medical education in response to COVID-19, summarizing developments and outcomes to serve as a guide for future work. The review highlighted areas of relative strength, as well as several gaps. Numerous articles have been written about remote learning and simulation and these areas are ripe for full systematic reviews. Telehealth, interviews and faculty development were lacking and need urgent attention.

A proximity-based graph clustering method for the identification and application of transcription factor clusters.

BACKGROUND: Transcription factors (TFs) form a complex regulatory network within the cell that is crucial to cell functioning and human health. While methods to establish where a TF binds to DNA are well established, these methods provide no information describing how TFs interact with one another when they do bind. TFs tend to bind the genome in clusters, and current methods to identify these clusters are either limited in scope, unable to detect relationships beyond motif similarity, or not applied to TF-TF interactions. METHODS: Here, we present a proximity-based graph clustering approach to identify TF clusters using either ChIP-seq or motif search data. We use TF co-occurrence to construct a filtered, normalized adjacency matrix and use the Markov Clustering Algorithm to partition the graph while maintaining TF-cluster and cluster-cluster interactions. We then apply our graph structure beyond clustering, using it to increase the accuracy of motif-based TFBS searching for an example TF. RESULTS: We show that our method produces small, manageable clusters that encapsulate many known, experimentally validated transcription factor interactions and that our method is capable of capturing interactions that motif similarity methods might miss. Our graph structure is able to significantly increase the accuracy of motif TFBS searching, demonstrating that the TF-TF connections within the graph correlate with biological TF-TF interactions. CONCLUSION: The interactions identified by our method correspond to biological reality and allow for fast exploration of TF clustering and regulatory dynamics.

A Signal Processing Approach for Detection of Hemodynamic Instability before Decompensation.

Advanced hemodynamic monitoring is a critical component of treatment in clinical situations where aggressive yet guided hemodynamic interventions are required in order to stabilize the patient and optimize outcomes. While there are many tools at a physician's disposal to monitor patients in a hospital setting, the reality is that none of these tools allow hi-fidelity assessment or continuous monitoring towards early detection of hemodynamic instability. We present an advanced automated analytical system which would act as a continuous monitoring and early warning mechanism that can indicate pending decompensation before traditional metrics can identify any clinical abnormality. This system computes novel features or bio-markers from both heart rate variability (HRV) as well as the morphology of the electrocardiogram (ECG). To compare their effectiveness, these features are compared with the standard HRV based bio-markers which are commonly used for hemodynamic assessment. This study utilized a unique database containing ECG waveforms from healthy volunteer subjects who underwent simulated hypovolemia under controlled experimental settings. A support vector machine was utilized to develop a model which predicts the stability or instability of the subjects. Results showed that the proposed novel set of features outperforms the traditional HRV features in predicting hemodynamic instability.

Digitization of Electrocardiogram From Telemetry Prior to In-hospital Cardiac Arrest: A Pilot Study.

BACKGROUND: Analyzing telemetry electrocardiogram (ECG) data over an extended period is often time-consuming because digital records are not widely available at hospitals. Investigating trends and patterns in the ECG data could lead to establishing predictors that would shorten response time to in-hospital cardiac arrest (I-HCA). This study was conducted to validate a novel method of digitizing paper ECG tracings from telemetry systems in order to facilitate the use of heart rate as a diagnostic feature prior to I-HCA. METHODS: This multicenter study used telemetry to investigate full-disclosure ECG papers of 44 cardiovascular patients obtained within 1 hr of I-HCA with initial rhythms of pulseless electrical activity and asystole. Digital ECGs were available for seven of these patients. An algorithm to digitize the full-disclosure ECG papers was developed using the shortest path method. The heart rate was measured manually (averaging R-R intervals) for ECG papers and automatically for digitized and digital ECGs. RESULTS: Significant correlations were found between manual and automated measurements of digitized ECGs (p < .001) and between digitized and digital ECGs (p < .001). Bland-Altman methods showed bias = .001 s, SD = .0276 s, lower and upper 95% limits of agreement for digitized and digital ECGs = .055 and -.053 s, and percentage error = 0.22%. Root mean square (rms), percentage rms difference, and signal to noise ratio values were in acceptable ranges. CONCLUSION: The digitization method was validated. Digitized ECG provides an efficient and accurate way of measuring heart rate over an extended period of time.

Automated Detection of Benzodiazepine Dosage in ICU Patients through a Computational Analysis of Electrocardiographic Data.

To enable automated maintenance of patient sedation in an intensive care unit (ICU) setting, more robust, quantitative metrics of sedation depth must be developed. In this study, we demonstrated the feasibility of a fully computational system that leverages low-quality electrocardiography (ECG) from a single lead to detect the presence of benzodiazepine sedatives in a subject's system. Starting with features commonly examined manually by cardiologists searching for evidence of poisonings, we generalized the extraction of these features to a fully automated process. We tested the predictive power of these features using nine subjects from an intensive care clinical database. Features were found to be significantly indicative of a binary relationship between dose and ECG morphology, but we were unable to find evidence of a predictable continuous relationship. Fitting this binary relationship to a classifier, we achieved a sensitivity of 89% and a specificity of 95%.